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Abstract— We  present  a  practical  approach  to  internet  routing  that 
provides  near-minimum  delays  over  multiple  loop-free  paths  to 
destinations.  The  new  protocol,  which  we  call  NEAR-OPT,  obtains 
multiple  loop-free  paths  to  destinations  using  long-term  delay  mea¬ 
sures,  and  allocates  destination-oriented  flows  over  such  paths  us¬ 
ing  short-term  delay  measures  to  minimize  delay.  We  compare  the 
performance  of  NEAR-OPT  with  traditional  single-path  routing 
and  the  only  known  adaptation  for  dynamic  networks  of  Gallager’s 
minimum-delay  routing  algorithm.  Using  actual  Internet  traffic 
traces  and  other  traffic  source  models,  we  show  that  NEAR-OPT 
provides  delays  comparable  to  the  lower  bounds  achievable  with 
Gallager’s  algorithm  for  static  networks,  provides  lower  delays 
than  implementations  of  Gallager’s  algorithm  in  networks  subject 
to  fractal  traffic,  and  renders  far  smaller  delays  and  better  use  of 
resources  than  traditional  single-path  routing.  NEAR-OPT  does 
not  depend  on  any  global  constant  and  is  completely  distributed, 
making  it  easy  to  implement  as  a  loop-free  ’’distance-vector”  pro¬ 
tocol  similar  to  Cisco’s  EIGRP. 

I.  Introduction 

Congestion  at  the  links  and  in  the  routers  is  the  main  cause  of  large 
delays  in  the  Internet.  All  the  Internet  routing  protocols  in  use  today 
rely  on  single-path  routing  algorithms  which  not  only  under-utilize  re¬ 
sources,  but  also  cannot  cope  with  congestion  as  all  traffic  for  a  desti¬ 
nation  should  be  routed  through  a  single  successor  and  when  that  link 
becomes  congested  its  whole  traffic  has  to  be  rerouted.  If  link  costs  are 
made  functions  of  congestion  or  delays  in  order  to  support  QoS  rout¬ 
ing,  routing-table  entries  can  become  unstable  in  single-path  routing 
protocols  [2],  [3]. 

To  diffuse  congestion  and  minimize  delays,  the  routing  protocol 
must  provide  multiple  paths  for  each  destination  at  each  router  that 
are  loop-free  at  every  instant  and  use  link  costs  that  are  a  function  of 
congestion  at  the  links.  Loop-free  routing  paths  are  crucial  because 
looping,  even  temporary  looping,  renders  longer  delays. 

Many  optimal  routing  algorithms  exist,  but  they  all  assume  the  input 
traffic  and  the  network  topology  to  be  stationary  or  very  slowly  chang¬ 
ing,  and  require  global  constants  that  guarantee  convergence.  This 
makes  optimal-routing  algorithms  impractical  for  internetworking,  be¬ 
cause  Internet  traffic  is  very  bursty  at  any  time  scale,  the  Internet  topol¬ 
ogy  may  experience  changes,  and  defining  global  constants  that  work 
for  all  input  traffic  patterns  are  impossible  to  determine. 

Gallager  [7]  proved  the  necessary  and  sufficient  conditions  for 
minimum-delay  routing  and  described  a  distributed  multipath  algo¬ 
rithm  that  is  loop-free  at  every  instant  and  obtains  minimum  delays 
when  input  traffic  and  the  network  topology  are  stationary  or  slowly 
changing  (quasi-static).  The  basic  result  shows  that,  to  avoid  oscil¬ 
latory  behavior  and  to  achieve  minimum  delays,  the  routing  protocol 

0  This  work  was  supported  in  part  by  the  Defense  Advanced  Research  Projects  Agency  ( DARPA)  under 
grants  F30602-97-1-0291  and  F19628-96-C-0O38. 


must  provide  multiple  paths  for  each  destination  at  each  router  that 
are  loop-free  at  every  instant  and  use  marginal(incremental)  delays  as 
dynamic  link  costs  and  it  must  incrementally  shift  traffic  from  con¬ 
gested  links  to  less  congested  links.  Segall  and  Sidi  [11],  [12]  ex¬ 
tended  Gallager’s  minimum-delay  routing  algorithm  to  handle  topo¬ 
logical  changes  using  techniques  developed  by  Merlin  and  Segall  [9]. 
In  this  paper,  we  call  this  algorithm  GSS  and  compare  it  against  our 
approach.  These  algorithms,  however,  are  not  suited  for  practical  use 
in  the  Internet  because  of  two  critical  requirements:  speed  of  conver¬ 
gence  and  relative  insensitivity  of  performance  to  variations  in  input 
traffic.  In  Bertsekas  and  Gallager  [1],  an  improved  version  of  Gallager 
is  presented  that  converges  quicker,  but  the  algorithm  is  still  limited  to 
small  networks  due  to  dependence  on  global  constants. 

Our  basic  design  approach  in  NEAR-OPT  protocol  consists  of  first 
establishing  multiple  loop-free  paths  to  a  destination  at  each  router  us¬ 
ing  long-term  delay  information,  and  then  allocating  flows  to  the  desti¬ 
nation  over  next-router  choices  available  at  each  router  for  a  destination 
using  short-term  delay  information.  The  complexity  of  implementing 
NEAR-OPT  is  similar  to  the  complexity  of  routing  protocols  that  pro¬ 
vide  single-path  routing. 

Section  II  formulates  the  minimum-delay  routing  problem.  Sec¬ 
tion  III  discusses  Gallager’s  minimum-delay  routing  algorithm  and  its 
drawbacks.  Section  IV  describes  NEAR-OPT  routing  which  adapts 
techniques  in  Gallager’s  algorithm  to  the  Internet  while  overcoming 
its  drawbacks.  Section  V  present  simulation  results  that  demonstrate 
the  advantages  of  NEAR-OPT  over  single-path  routing  and  Gallager’s 
algorithm  under  dynamic  traffic  conditions. 

II.  Problem  formulation  and  background 

The  minimum-delay  routing  problem  can  be  formulated  as  follows 
[7].  A  computer  network  G  =  ( N,L )  is  made  up  of  N  routers  and  L 
links  between  them.  Each  link  is  bidirectional  with  possibly  different 
costs  in  each  direction. 

Let  r'j  >  0  be  the  expected  input  traffic,  measured  in  bits  per  second, 
entering  the  network  at  router  i  and  destined  for  router  j.  Let  tj  be  the 
sum  of  r':i  and  the  traffic  arriving  from  the  neighbors  of  i  for  destination 
j.  Finally,  let  routing  parameter  ’ k  be  the  fraction  of  traffic  tj  that 
leaves  router  i  over  link  (i,  k).  Assume  that  the  network  does  not  lose 
any  packets.  Then,  from  conservation  of  traffic  we  have 

C  L  +  X!  ^  A  (1) 

keN‘ 

where  Nl  is  the  set  of  neighbors  of  router  i. 

Let  fik  be  the  expected  traffic,  measured  in  bits  per  second,  on  link 
( i ,  k).  Because  is  the  traffic  destined  for  router  j  on  link  (i,  k) 

we  have  the  following  equation  to  find  /,/, . 
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Note  that  0  <  /,*  <  Cik  where  Cik  is  the  capacity  of  link  ( i ,  k)  in 
bits  per  second. 


Property  1:  For  each  router  i  and  destination  j,  the  routing  parameters 
0'j k,  satisfy  the  following  conditions: 

1.  4>jk  =  0  if  (i,  k)  (£  L  or  i  =  j.  Clearly  if  the  link  does  not  exist, 
there  will  be  no  traffic  on  it. 

2-  4>)k  —  0-  Negative  amount  of  traffic  cannot,  of  course,  be  allo¬ 
cated. 

3.  y\  fjj  d)jk  =  1.  All  traffic  must  be  allocated. 

Let  Du-  be  defined  as  the  expected  number  of  messages  or  packets 
per  second  transmitted  on  link  (i,k)  times  the  expected  delay  per  mes¬ 
sage  or  packet,  including  the  queuing  delays  at  the  link.  We  assume 
that  messages  are  delayed  only  by  the  links  of  the  network  and  Djk 
depends  only  on  flow  /;,  through  link  (j,  k )  and  link  characteristics 
such  as  propagation  delay  and  link  capacity.  Djk(fik)  is  a  continuous 
and  convex  function.  The  total  expected  delay  per  message  times  the 
total  expected  number  of  message  arrivals  per  second  is  given  by 


Dt=  Dik(fik)  (3) 

(i,k)eL 
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Equation  (4)  shows  the  relation  between  node’s  marginal  distance 
and  the  marginal  distances  of  neighbors  to  a  particular  destination. 
Equations  (5),  (6)  and  (7)  indicate  that  under  peifect  load  balanc¬ 
ing,  ie.,  when  routing  parameter  set  <j>  gives  the  minimum  delay,  the 
marginal  distances  through  neighbors  in  the  successor  set  are  equal, 
and  the  marginal  distances  through  neighbors  not  in  the  successor  set 
are  higher  than  those  in  the  successor  sets.  Let  D]  denote  the  marginal 
distance  dDr/dr]  from  i  to  j.  Let  l\  denote  the  marginal  delay 
Djkifik)  as  the  cost  of  the  link  from  i  to  j.  Let  S]  be  the  set  of 
neighbors  through  which  router  i  forwards  traffic  towards  j.  Now  the 
minimum  delay  routing  problem  becomes  one  of  determining  routing 
parameters  { <!>] k  }  at  each  node  i  for  each  destination  j,  such  that  the 


following  equations 

are  satisfied. 

D) 

=  <t>] k(Dj  +  I'k) 

keN * 

(8) 

s) 

=  {k\4>)k  >  0  Afc  G  N*} 

(9) 

D) 

<  Dj  +l{  k  G  JV! 

(10) 

w+i) 

=  (-d;j  +  4) 

(11) 

(D*+Q 

<  (D«  +  g  pes]  q$S) 

(12) 

Note  that  the  router  traffic  flow  set  t  =  [t] }  and  link  flow  set 
/  =  {/,, }  can  be  obtained  from  r  =  {r]}  and  <j>  =  Dt 

can  therefore  be  expressed  as  a  function  of  r  and  (j)  using  (1)  and  (2). 
The  minimum-delay  routing  problem  can  now  be  stated  as  follows:  For 
a  given  fixed  topology  with  link  capacities  Cik  and  input  traffic  flow 
set  r,  and  delay  function  7?;, (/,:,)  for  each  link  (j,  k),  the  minimiza¬ 
tion  problem  consists  of  computing  the  routing  parameter  set  <j>  such 
that  the  total  expected  delay  Dt  is  minimized. 

III.  A  Minimum  Delay  Algorithm 

To  solve  the  minimization  problem  described  in  the  previous  sec¬ 
tion,  Gallager  [7]  derived  the  necessary  and  sufficient  conditions  and 
described  an  algorithm  to  compute  routing  parameter  set  <j>  such  that 
those  conditions  are  satisfied.  To  find  the  conditions,  first  obtain  the 
partial  derivatives  of  the  total  delay  Dt  of  (  3)  with  respect  to  r  and  <j>, 
that  is. 


dPT 

Or] 

ODt 

d*jk 
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(4) 
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where  D'  ,,  (/;,)  =  an[j  |s  called  as  marginal  or  incremental 

delay.  is  the  marginal  distance  from  node  i  to  j.  The  Gallager 

3 

theorem  is  stated  here  for  discussion. 

Theorem  1:  ( Gallager[7 ]):  The  necessary  condition  for  a  minimum 
of  Dt  with  respect  to  <f>  for  all  i  ^  j  and  (i,k)  G  L  is 


Gallager  [7]  describes  a  distributed  routing  algorithm  for  stationary 
and  quasi-stationary  networks  based  upon  the  above  equations.  Segall 
and  Sidi  [11].  [12],  extended  Gallager’ s  basic  algorithm  to  handle  topo¬ 
logical  changes.  We  refer  to  this  algorithm  as  GSS.  A  detailed  descrip¬ 
tion  of  Gallager’s  algorithm  and  its  extensions  handling  topological 
changes  are  presented  in  [7],  [11].  GSS,  however,  is  not  practical  for 
several  reasons.  A  major  drawback  of  GSS  is  that  a  global  step  size  p 
needs  to  be  chosen  and  every  router  must  use  it  to  ensure  convergence; 
because  this  constant  depends  on  the  traffic  pattern,  it  is  impossible 
to  determine  one  in  practice  that  works  for  all  input  traffic  pattern. 
In  GSS,  the  routing  parameters  are  directly  computed  and  the  multi¬ 
ple  loop-free  paths  are  indirectly  implied  by  the  routing  parameters  in 
(9).  The  computation  of  routing  parameters  in  GSS  is,  for  all  practi¬ 
cal  purposes,  a  very  slow  process  because  it  is  a  destination-controlled 
process;  i.e.,  the  destination  must  initiate  every  iteration  that  adjusts  the 
routing  parameters  at  every  router,  making  the  algorithm  slow  converg¬ 
ing  and  useful  only  for  quasi-static  routing,  where  traffic  and  topology 
are  static  long  enough  for  all  routers  to  adjust  their  routing  parame¬ 
ters  between  changes.  In  a  network  with  a  large  diameter,  each  itera¬ 
tion  will  take  a  time  proportional  to  the  diameter  of  the  network  and 
a  number  of  messages  proportional  to  number  of  links.  Depending 
on  the  global  constant  r],  several  iterations  are  needed  to  converge  to 
the  final  routing  tables.  The  number  of  iterations  needed  for  conver¬ 
gence  tends  to  be  large  for  a  small  r],  and  small  for  a  large  value  of 
r).  However,  p  cannot  be  made  arbitrarily  large  to  reduce  number  of 
iterations  and  to  speed  up  convergence,  because  the  algorithm  may  not 
converge  at  all  for  large  p.  Hence,  GSS  is  basically  a  method  for  ob¬ 
taining  lower  bounds  under  stationary  traffic  rather  than  an  algorithm 
for  use  in  practical  networks.  In  the  next  section,  we  show  how  the 
theory  and  technique  available  in  the  Gallager  method  can  be  adapted 
to  practical  networks,  while  overcoming  its  drawbacks. 


dDr  =  J  =  Ay  >  0 

Oo]k  \  >  Ay  o',  0 


(6) 


where  A,:j  is  some  positive  number,  and  the  sufficient  condition  to  min¬ 
imize  Dt  with  respect  to  <fi  is  for  all  i  ^  j  and  [i,k)  G  L  is 


IV.  Near-optimum  Delay  Routing 

We  noted  that  in  GSS  the  computation  of  the  routing  parameter  set  <j> 
was  slow  converging  and  works  only  in  the  case  of  stationary  or  quasi- 
stationary  traffic.  In  the  Internet,  traffic  is  hardly  stationary  and  per¬ 
fect  load  balancing  is  neither  possible  nor  necessary.  An  approximate 


load  balancing  scheme  based  on  some  heuristic  which  can  be  quickly 
adapt  to  dynamic  traffic  is  actually  sufficient  to  minimize  delays  sub¬ 
stantially.  The  key  idea  in  our  approach  is  that  multiple  paths  ar e  first 
computed  by  other  means  and  then  routing  parameters  are  assigned  to 
each  successor  using  some  heuristics,  i.e.,  the  process  of  computing 
successor  sets  is  decoupled  from  computation  of  routing  parameters. 
This  gives  NEAR-OPT  the  ability  to  quickly  respond  to  traffic  bursts 
using  short-term  metrics  while  providing  shortest  paths  based  on  long- 
term  metrics.  So  in  our  approach,  we  approximate  (8)  to  (12)  with  the 
following: 


D) 

=  min{Dj  +  l’k\k  £  Ar!) 

(13) 

si 

=  {k\Dj  <  Dj  Ak  £  AT*} 

(14) 

=  >M  .  ,1  .  II  )  k  £  A 

(15) 

where  A)  =  !/);'  +  Z‘| p  £  JV’}  and  Bj  =  {<j>)p\p  £  Ni}.  We 
observe  that  (13)  is  the  Distributed  Bellman-Ford  (DBF)  equation  for 
computing  the  shortest  path  and  (14)  is  the  successor  set  that  includes 
the  neighbors  that  are  closer  to  the  destination  than  the  node  itself. 
Note  that  this  automatically  includes  the  neighbor  that  offers  the  short¬ 
est  path.  The  function  'T  in  (15)  is  a  heuristic  function  that  finds  the 
routing  parameters. 

It  is  well  documented  that  routing  algorithms  based  on  DBF  suffer 
from  severe  performance  problems  due  to  count-to-infinity  and  looping 
problems  [13].  To  confront  these  problems  NEAR-OPT  uses  diffusing 
computations  first  described  by  Dijkstra-Scholten[5],  and  extends  the 
basic  mechanism  to  permit  multiple  successors  per  node  for  each  desti¬ 
nation.  Because  of  space  limitation,  to  see  how  NEAR-OPT  computes 
Dj  and  ,S'j ,  the  reader  is  referred  to  a  detailed  description  and  correct¬ 
ness  proof  of  an  algorithm  called  DASM  [14].  To  understand  the  key 
point  of  this  paper  it  is  sufficient  to  know  that  NEAR-OPT  computes 
Sj  such  that  the  successor  graph  to  the  destination  j  implied  by  them 
is  loop-free  at  every  instant. 

A.  Handling  Network  Congestion 

With  variable  traffic,  the  flows  over  links  are  continuously  changing, 
causing  continuous  link  cost  changes.  To  respond  to  these  changes, 
queuing  delays  at  the  links  should  be  measured  periodically  and  rout¬ 
ing  paths  recomputed  accordingly.  However,  frequent  recomputation 
of  routing  paths  consumes  excessive  bandwidth  and  computation  re¬ 
sources  because  of  which  routing  path  changes  should  only  be  done 
at  sufficiently  long  intervals.  Unfortunately,  a  network  cannot  be  re¬ 
sponsive  to  short-term  traffic  bursts  if  only  long-term  updates  are  used. 
Therefore,  NEAR-OPT  performs  two  types  of  link-cost  updates:  short¬ 
term  updates  which  are  made  every  Ts  seconds,  and  long-term  updates 
which  are  made  every  Ti  seconds.  In  general  Ts  <<  T; .  The  long-term 
updates  are  designed  to  handle  long-term  traffic  changes  and  result  in 
updating  the  successor  sets  at  each  router,  such  that  the  new  routing 
paths  are  the  shortest  paths. 

The  short-term  updates  made  every  Ts  seconds  are  designed  to  han¬ 
dle  short-term  traffic  fluctuations  that  occur  between  long-term  routing 
path  updates  and  are  used  to  compute  the  routing  parameters  cj>jk  in 
(15)  at  each  router;  successor  sets  and,  therefore,  successor  graphs  do 
not  change.  Since  traffic  allocation  heuristic  in  (15)  is  a  function  of  the 
congestion  at  the  links  it  effectively  reduces  traffic  on  congested  links 
and  increases  traffic  on  less  congested  links.  in  general  can  be  any 
function  that  satisfies  Property  1,  but  to  get  maximum  performance  it 
must  be  made  a  function  of  the  marginal  distances  through  the  succes¬ 
sors.  Various  heuristics  can  be  used  to  compute  the  routing  parameters; 
we  present  one  such  heuristic  later  in  this  section. 

Unlike  r/  in  GSS,  T)  and  Ts  are  local  constants  that  are  set  indepen¬ 
dently  at  each  router.  Also,  they  need  not  be  static  constants  and  can  be 
made  to  vary  according  to  level  of  congestion  at  the  node.  The  value 


Algorithm  IH 

(1)  For  each  k  £  Sj  set 

<4  <  0. 

(2)  If  \Sj  |  =1.  then  for  k  £  Sj  set 
(l>)k  <r-  1. 

(3)  If  |  Sj  |  >  1,  then  for  each  k  €  Sj  set 

D‘  +!’ 

_ jfc  fc _ 

E.  (d\  +i*_) 

me  S\  3-m  m 

Af  j  _ -1 _ 

cjk  ^  (is1*  |  —  i) 

End  IH 

Fig.  1 .  Heuristic  for  initial  load  assignment. 


of  T),  however,  should  be  such  that  it  is  sufficiently  longer  than  the 
time  it  takes  for  NEAR-OPT  to  compute  shortest  paths.  The  long-term 
update  periods  should  be  phased  randomly  at  each  router,  because  of 
the  problems  that  would  result  due  to  synchronization  of  updates  [2], 

In  the  minimization  problem  in  Section  III  the  link-cost  metric  used 
is  the  marginal  delay ,  which  is  the  rate  of  change  of  delay  with  respect 
to  flow  through  the  link.  That  is,  if  D,k  (/,/, )  is  the  delay  of  link  (i,  k ) 
as  a  function  of  flow  in  the  link,  then  the  marginal  delay  is  D'ik  ( fik )  = 
■  In  NEAR-OPT.  therefore,  we  use  marginal  delays  for  link  costs. 

For  the  purposes  of  simulations  and  providing  intuition  we  have  ap¬ 
proximated  the  link  behavior  to  an  M/M/1  queue.  We,  therefore, 
obtain  the  following  formula  for  marginal  delay  at  a  link  [2],  where  r,;* 
is  the  propagation  delay  of  link  ( i,k ). 

D\k(fik )  =  7 -=  Ci\  y2  +  rik  (16) 

(Cik  -  Jik) 

Although  (16)  is  useful  for  describing  the  intuition  in  NEAR-OPT, 
it  is  limited  because  links  are  hardly  M/M/1  in  real  networks  due  to 
the  characteristics  of  the  traffic  over  the  links.  Accordingly,  in  real  im¬ 
plementation  of  our  approach,  the  marginal  delay  of  a  link.  D[k  (fik), 
should  be  computed  on-line  by  measuring  traffic  and  the  delays  at  the 
link.  Segall  [11]  and  Cassandras  [4]  suggests  some  practical  techniques 
for  estimating  D'ik  {fik)-  Average  link  costs  are  measured  over  two  dif¬ 
ferent  intervals;  link  costs  measured  over  short  period  Ts  are  used  for 
routing  parameter  computation  and  link  costs  measured  over  long  pe¬ 
riod  T;  are  used  for  routing  path  computation. 

B.  Assigning  Flows  to  Successor  Sets 

We  now  describe  how  (pj  k  in  (15)  are  computed  using  1P.  When 
Sj  is  computed  for  the  first  time  or  recomputed  again  due  to  long- 
term  route  changes,  traffic  should  be  freshly  distributed  among  the 
successors.  In  this  case,  since  traffic  is  not  already  distributed  the  al¬ 
location  heuristic  function  tp  is  a  function  of  only  the  marginal  dis¬ 
tances  through  the  successor  set.  That  is  (15)  reduces  to  the  form 
Wjk)  =  ^{k-{Dj  +  l’p\p  £  A”}).  When  a  new  successor  set  Sj 
is  computed,  algorithm  IH  in  Fig.  1  is  first  used  to  distribute  traf¬ 
fic  over  the  successor  set.  Note  that  { (pj k }  computed  in  IH  satisfy 
Property  1.  Furthermore,  when  more  than  one  successor  is  present,  if 
Djp  +  Ij,  >  Djq  -f  l'q  for  successors  p  and  q,  then  <pjp  <  <pjq.  The 
heuristic  makes  sense  because  the  greater  the  marginal  delay  through 
a  particular  neighbor  becomes,  the  smaller  the  fraction  of  traffic  that  is 
forwarded  to  that  neighbor. 

After  the  first  flow  assignment  is  made  over  a  newly  computed  suc¬ 
cessor  set  using  algorithm  IH,  a  different  flow  allocation  heuristic  algo¬ 
rithm  AH  shown  in  Fig.  2  is  used  to  adjust  the  routing  parameters  every 
Ts  seconds  until  the  successor  set  changes  again.  The  heuristic  func¬ 
tion  computed  in  AH  is  incremental  and,  unlike  IH,  is  a  function  of 
current  flow  allocation  on  the  successor  sets  and  the  marginal  distances 


Algorithm  AH 

(1)  Find  Dliin  <-  min{D)k  +  |ft  G  5]}. 

Let  ko  be  the  neighbor  that  offers  this  minimum. 

That  is  (O>0+/:,). 

(2)  Let  a)k  <—  D)k  +  l\  -  D»in  for  each  k  G  S). 

(3)  Let  A  <—  ^min{~-\k  G  S)  A  a‘]k  ±  0} 

j  fc 

(4)  For  each  ft  ft0  A  ft  G  set  <j>jk  G  ^  -Ax  a’jk 

(5)  For  ft  =  ft0  set  <j>)k  <—  <j>)k  +  A  x  ajg- 

End  AH 


Fig.  2.  Heuristic  for  incremental  load  adjustment. 


Fig.  3.  The  network  models  used  in  simulations 


through  the  successors.  AH  also  preserves  property  1  at  every  instant. 
In  AH  traffic  is  incrementally  moved  from  the  links  with  large  marginal 
delays  to  links  with  the  least  marginal  delay.  The  amount  of  traffic 
moved  away  from  a  link  is  proportional  to  how  large  the  marginal  de¬ 
lay  of  the  link  is  compared  to  the  best  successor  link.  The  heuristic 
tends  to  eventually  distribute  traffic  in  such  a  way  that  the  marginal 
delays  of  all  the  successors  are  equal  which  is  indeed  the  necessary 
condition  in  the  Gallager  theorem. 

V.  Simulations 

In  this  section,  we  address  the  average  performance  of  NEAR-OPT, 
which  is  a  direct  consequence  of  its  use  of  multiple  loop-free  paths 
and  the  flow  allocation  heuristic.  We  performed  a  series  of  simula¬ 
tions  to  compare  the  delay  and  throughput  of  NEAR-OPT,  GSS  and  a 
single-path  routing  algorithm  DUAL[8]  under  identical  settings.  We 
chose  DUAL  for  comparative  study  because  it  provides  loop-free  sin¬ 
gle  paths  at  every  instant  and  it  is  an  algorithm  being  used  in  an  ex¬ 
isting  Internet  routing  protocol,  EIGRP  [6].  Furthermore,  DUAL  has 
been  shown  to  converge  faster  than  the  traditional  topological  broad¬ 
cast  protocols  (e.g.,  OSPF)  after  link-cost  changes  [13],  and  constitutes 
a  better  single-path  routing  approach  with  which  to  compare  NEAR- 
OPT. 

The  network  topology  models  for  which  we  present  simulation  re¬ 
sults  in  this  paper  are  shown  in  Fig.  3.  These  small  network  topology 
was  chosen  to  speed  up  simulations  while  not  favoring  NEAR-OPT, 
and  have  some  similarities  with  the  sparse  connectivity  of  routers  in 
the  Internet.  All  the  links  have  the  same  link  capacity  of  1.0 Mbs.  The 
propagation  delay  of  the  links  is  lOps.  The  long-term  update  period  T) 
is  set  at  1.0s,  while  the  short-term  update  period  Ts  is  set  at  0.1s.  We 
used  (16)  for  the  marginal  delay  in  our  simulations  for  simplicity  and 
it  does  not  favor  any  particular  algorithm. 

A.  Experiment  I 

In  this  experiment  we  compare  NEAR-OPT,  GSS  and  DUAL  under 
stationary  traffic.  The  flows  are  exponential  with  constant  rate.  Flow-0 
starts  at  router  9  and  is  destined  for  router  0.  Flow-1  starts  at  router  8 
and  is  received  at  router  1.  Flow-2  and  flow-3  are  intervening  flows. 
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Experiment  I 


Fig.  4.  Avg.  delays  under  stationary  traffic 


Experiment  II 


Fig.  5.  Avg.  delays  under  variable  traffic 


that  start  at  routers  5  and  4  respectively  and  are  received  at  routers  0 
and  1,  respectively.  The  flow  in  the  link  is  obtained  by  measuring  the 
number  of  bits  (data  and  control)  that  are  transmitted  on  the  link  in  a 
time  interval.  The  link  capacities,  propagation  delays  and  data  packet 
sizes  are  kept  constant  and  are  the  same  for  all  algorithms. 

The  delays  obtained  for  NEAR-OPT,  GSS  and  DUAL  for  flow-0 
are  shown  in  the  Fig.  4.  We  observe  that  NEAR-OPT  and  GSS  signif¬ 
icantly  outperform  DUAL,  because  of  the  use  of  multiple  paths.  We 
see  that  GSS  performs  better  than  NEAR-OPT,  but  the  delays  from 
NEAR-OPT  are  within  20%  of  those  of  GSS.  An  important  observa¬ 
tion  from  this  experiment  is  that  the  response  of  NEAR-OPT  to  a  step 
function  is  far  better  than  GSS.  In  this  case,  the  step  function  consists 
of  starting  a  given  flow  at  time  0.  The  reason  NEAR-OPT  has  a  much 
less  underdamped  response  to  drastic  changes  in  input  traffic  is  that  the 
propagation  of  routing  information  is  much  faster  in  NEAR-OPT  than 
in  GSS.  Another  observation  is  that  the  difference  in  delays  obtained  in 
NEAR-OPT  and  GSS  after  GSS  attains  the  initial  flow  assignments  on 
the  routers  is  always  smaller  than  the  initial  deficit  in  GSS.  This  pre¬ 
dicts  that  NEAR-OPT  should  have  a  favorable  performance  over  GSS 
when  either  traffic  is  very  bursty  or  the  topology  experiences  drastic 
changes.  Note  that  NEAR-OPT's  performance  compared  to  DUAL 
can  be  even  better  if  the  connectivity,  ie.,  the  average  node  degree  is 
higher,  because  then  NEAR-OPT  can  use  more  successors. 
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Experiment  IV 


Fig.  6.  Throughput  comparison 


Fig.  7.  Avg.  delays  for  flow-0  under  internet  traffic 


B.  Experiment  II 

This  experiment  studies  the  responsiveness  of  the  protocols  under 
bursty  traffic  by  repeating  Experiment  I  using  ON-OFF  traffic  pattern 
for  the  flows.  The  delays  obtained  for  flow-0  are  shown  in  Fig.  5  for 
NEAR-OPT,  GSS  and  DUAL.  We  observe  that  GSS  and  NEAR-OPT 
again  outperformed  DUAL  by  virtue  of  multiple  paths.  NEAR-OPT’s 
better  performance  under  variable  traffic  is  due  to  the  slower  propa¬ 
gation  of  long-term  path  information  in  GSS.  Note  that  the  network 
on  which  this  simulation  is  performed  is  small;  in  a  more  realistic 
topology  the  slow  convergence  of  GSS  becomes  more  pronounced  and 
comparatively  NEAR-OPT  performs  even  better  than  this  experiment 
shows.  This  experiment  indicates  that  the  basic  intuition  in  NEAR- 
OPT’s  design  is  correct,  i.e.,  building  loop-free  paths  using  long-term 
delay  information  first  and  then  assigning  flows  based  on  heuristics  is 
appropriate,  because  the  latency  of  making  routers  agree  on  loop-free 
paths  dominates  the  latency  of  the  local  flow  assignment  decisions  at 
each  router. 

C.  Experiment  III 

In  this  experiment,  we  measure  throughput  of  NEAR-OPT,  GSS  and 
DUAL.  The  throughput  of  the  network  is  the  total  data  delivered  by 
all  flows  in  the  network  in  a  given  time  interval.  Fig.  3(b)  shows  the 
network  and  the  flows  used  in  this  experiment.  We  use  smaller  network 
because  it  is  easy  to  saturate  the  network.  The  flows  use  ON-OFF 
traffic  with  exponential  distribution  during  the  ON  period.  Fig.  6  shows 
that,  for  a  given  delay,  a  much  greater  throughput  and  hence  greater 
network  utilization  can  be  achieved  in  NEAR-OPT  than  in  GSS  and 
DUAL. 

D.  Experiment  IV 

This  experiment  repeats  experiment  II,  but  uses  actual  Internet  traf¬ 
fic  traces  obtained  from  Lawrence  Berkeley  National  Laboratory  [10]. 
Fig.  7  shows  the  results  when  traffic  traces  extracted  from  traffic  trace 
LBL-PKT-4  are  used  for  flows  0,  1,  2  and  3.  We  can  see  that  NEAR- 
OPT  outperforms  GSS  and  DUAL  which  is  expected  from  the  results  in 
Experiment  II  and  III.  because  traffic  is  only  made  more  bursty  in  this 
experiment.  This  points  to  the  fact  that  GSS  is  basically  a  method  for 
obtaining  lower  bounds  under  stationary  traffic,  and  that  NEAR-OPT  is 
fast  enough  to  take  advantage  of  multiple  paths  with  very  bursty  traffic 
and  can  dramatically  increase  performance  in  current  internets. 


VI.  Conclusions 

We  have  presented  a  practical  approach  to  near-minimum  delay 
routing  in  computer  networks  and  in  the  Internet.  We  first  described 
Gallager’s  method  to  derive  the  lower  bounds  for  delays  that  are  the¬ 
oretically  possible.  We  pointed  out  the  drawbacks  of  the  algorithms 
directly  based  on  Gallager’s  method,  namely,  slow  convergence  and 
requirement  of  a  global  constant  r]  for  ensuring  convergence  and  recti¬ 
fied  them  in  our  algorithm. 

Simulation  results  show  that  NEAR-OPT  performs  significantly  bet¬ 
ter  than  single-path  routing,  and  that  NEAR-OPT  performance  comes 
within  a  small  percentage  range  of  the  lower  bound  delays  under  sta¬ 
tionary  traffic,  and  that  under  varying  traffic  conditions,  NEAR-OPT 
performs  better  than  GSS. 

In  closing,  it  must  be  emphasized  that,  although  theoretically  min¬ 
imum  delays  cannot  be  achieved  in  practice  simply  because  of  the  la¬ 
tencies  involved  in  propagation  of  routing  information,  NEAR-OPT 
demonstrates  that  far  better  performance  can  be  obtained  using  practi¬ 
cal  approaches  similar  to  those  already  used  in  the  Internet  for  single¬ 
path  routing. 
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